{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ❇️ Automatic differentiation in PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Back propagation is widely used for training neural nets. During back propagation we adjust model<br> parameters (weights & biases) based on the gradient of the loss function w.r.t. the given parameter.<br> ∂loss/∂w (gradient w.r.t weight w); ∂loss/∂b (gradient w.r.t. bias b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 🔴 Let's consider a simplest neural net with 3 inputs and 2 outputs 👇 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](./resources/basic_neural_net.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 🟡 Let's code ⬆️  this using pytorch ⬇️ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "x = torch.ones(3)  # input tensor\n",
    "y = torch.zeros(2)  # expected output\n",
    "# Notice the use of requires_grad = True ⬇️ \n",
    "w = torch.randn(3, 2, requires_grad=True) # weights \n",
    "b = torch.randn(2, requires_grad=True) # biases \n",
    "z = torch.matmul(x, w)+b # output\n",
    "loss = torch.nn.functional.binary_cross_entropy_with_logits(z, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above ⬆️ code represents the following ⬇️ computationl graph <br>\n",
    "In this graph, w and b are parameters, which we need to optimize. <br> \n",
    "Thus, we need to be able to compute the gradients of loss function with respect to those variables. <br>\n",
    "In order to do that, we set the <b>requires_grad</b> property of those tensors. <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](./resources/computational_graph.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 🟢 grad_fn:\n",
    "grad_fn is an object of class Function that is applied to tensors to construct computational graph ⬆️ .<br> This object knows how to compute the function in the forward direction, and also how to compute its <br>derivative during the backward propagation step.<br> grad_fn becomes property of a tensor. Check this out 👇 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "grad_function for z = <AddBackward0 object at 0x7fd9d92fe748>\n",
      "grad_function for loss = <BinaryCrossEntropyWithLogitsBackward object at 0x7fd9d92fe5c0>\n"
     ]
    }
   ],
   "source": [
    "print(f\"grad_function for z = {z.grad_fn}\")\n",
    "print(f\"grad_function for loss = {loss.grad_fn}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 🔵 Computing Gradients\n",
    "To optimize weights (w) of our neural network, we need to compute we need ∂loss/∂w (gradient of loss w.r.t weight w);<br> ∂loss/∂b (gradient of loss w.r.t. bias b) under some fixed values of x and y. <br>\n",
    "This is how we do it 👇 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.2980, 0.4528],\n",
      "        [0.2980, 0.4528],\n",
      "        [0.2980, 0.4528]])\n",
      "tensor([0.2980, 0.4528])\n"
     ]
    }
   ],
   "source": [
    "# loss.backward() Computes the gradient of current tensor w.r.t. graph leaves.\n",
    "# In the graph we see that the leaves are w and b (ones for which required_grad = True)\n",
    "loss.backward()\n",
    "print(w.grad)\n",
    "print(b.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "env_torch",
   "language": "python",
   "name": "env_torch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
